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Abstract 



The present concern over the number, quality, and 
appropriateness of surveys being used to collect research 
data, especially for policy analysis, has led to questions 
concerning the need for, and alternatives to, survey method- 
ologies. This paper presents an analysis of the drawbacks 
to the use of surveys in economic research and suggestions 
for ways they may be more profitably incorporated in future 
research. Examples of such innovative strategies from the 
economic research area are discussed. 
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I. 



Introduction 



There seems to be an increasing reliance on the use of 
surveys to collect data to be used in a wide variety of research 
settings. In many cases the research results are to be used to 
derive (or support) policy decisions. The questions which arise, 
and which motivate these remarks, are: "Is survey research 

necessary?"; "Is it cost-effective?"; "What alternatives exist?"; 
and "How might they be done better, when necessary?". 

II . Economic Research and Use of Survey Methods 

There exist two major ways economists analyse "what people 
do" in order to predict what their future economic behavior will 
be: 

a) They assume that there exists a utility function with 
certain properties which embody the important decision variables 
and parameters, and which relates their combination to levels of 
consumer satisfaction; then they predict on the basis of changes 
in parameters through analytical manipulation of mathematical 
models based on these functions. 

b) They assume that people behave rationally, or at least 
consistently, and observe what people do, or did, and therefore 
derive predictions of what they will do and how well off they will 
be, comparatively, when parameters change. This is the "revealed 
preference" approach. 

In the former case, we may use the theory of consumer be- 
havior so developed to construct testable hypotheses of the way 
they will behave in the future, and judge the "validity" of the 
underlying assumptions and theoretical structures (e.g., "forms" 
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of utility functions) on the basis of the actually observed be- 
havior. In this case one may argue that surveys taken before 
parametric changes in one's environment occur, or before decisions 
are made even without such changes, will provide "ex ante" in- 
formation which may enable the theoretical structures to be modified 
and thus be better predictors of results than if events are merely 
"allowed" to happen. 

However economists have been traditionally skeptical of the 
value of many such survey techniques because of their knowledge 
of the basic incentives which are believed to motivate behavior. 

For example, many experiments have demonstrated the failure of 
the "what if" type hypothetical choice results to conform to the 
ubiquitous utility theory assumption of "Transitivity of Preferences". 
The rationale is that since the subject consumers did not actually 
have to make and, live with, their choices, their incentives were 
insufficient to get them to make the true choices (i.e., the ones 
they really would have made in such an actual choice situtation) . 

An additional confounding factor occurs in situations where the 
goods or services under study are not pure "private goods" but 
rather exhibit varying degrees of externalities and/or are provided 
by the government (at some level) as "public goods". The problem 
here is simple: If the amount of a good or service to me is not 

diminished by your consumption of it, I'd rather that you pay for 
it, or that you be responsible for its existence rather than me. I 
have an incentive to hide (to different degrees) my true preferences 
in such cases. 

Next consider the revealed preference approach. If we wait 
until behavior is actually observable to attempt relative utility 
calculations [not between consumers, but for an average, or subset 
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of, consumer (s) ] , we have lost the ability to influence these 
utility levels. In addition, many other variables will have also 
changed, making it difficult to lay the credit or blame for changes 
in satisfaction on the targeted policy variable. Likewise if we 
collect only data from past actions, we may have too few data 
points from which to generalize and the ceteris paribus conditions 
may also have been violated. Thus although it would seem that 
surveys could facilitate collection of more data points and then 
enhance the probability of accurate predictions, even when based 
in large part on the revealed preference approach, the same ob- 
jections as noted above regarding the utility theory approach 
would then hold in this case. 

For these reasons economists have long been hesitant to rely 
in any significant way on survey data in the future sense of the 
word, that is, where hypothetical situations and questions are 
analysed. We as a profession have been more amenable to the use 
of surveys to collect data dealing with the present, i.e., factual 
augmentation, when files and records are insufficient. However, 
some economists certainly have based their research efforts on 
survey work. The recognized leader among economists in the develop- 
ment and use of survey data is George Katona, who until recently 
was research coordinator at the Survey Research Center at the 
University of Michigan. His position and proclivities have led 
him to differ significantly with the majority of economists in 
assessing the purpose and utility of survey research. He has said 
that "The primary purpose of surveys are to obtain information on 
trends over time and a functional relationship among variables. 
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This statement has, first, a negative implication: Determination 

of a position as it prevails at a given time is not the major 
purpose of surveys .... Such findings do not constitute major goals 
of survey research and are subject to large errors .... Absolute 
measurements represent but the starting point for studies in 
Behavioral Economics" [3, p. 409-410]. 

Katona ' s assertion that "Studies of changes in attitudes and 
expectations and their relation to behavior call for the collection 
of new data [through surveys]", is tempered by his further state- 
ment that "field observations and experiments may supplement the 
survey method" [p. 405]. The major assertion however is consistent 
with the pioneering work done over the years by the Survey Research 
Center in providing information on the attitudes and intentions of 
consumers and business decision makers which has proven extremely 
useful in formulating macro-economic policy. 

In my view the situation is not clear cut at all. As is 
usually the case, the use of survey versus non-survey methods to 
collect information and data in a variety of problem settings should 
be determined by the problem and its setting, and generally will in- 
volve some judiciously chosen mix of the two methodologies. Addition- 
ally one must distinguish between surveys which collect: 

a) Data on physically existing facts (e.g., the shelf- 
prices at stores) which do not exist in records or files; 

b) Data on non-manipulable individual characteristics 
(e.g. , demographics) ; 

c) Data which requires a person to recall behavior or 
opinions from the past; 

d) Data on probable future behavior or expectations. 



5 



Certainly the majority of economists have few qualms about 
the first two uses of surveys. For example, in my own research, 
we have surveyed the catchment population eligible to use military 
health care facilities to ascertain the age/sex beneficiary status 
in order to derive subgroup utilization rates. Such data does 
not exist in the presently structured military health care system. 

The problems with these sorts of surveys are mostly technical — 
i.e., selecting the correct sample size and its composition to ob- 
tain reasonable estimates, and are very amenable to solution. 

However there are more serious problems with the latter two 
types of surveys. Many times people cannot accurately recall their 
behavior or facts about a previous situation. Added to this are 
the incentive problems mentioned above associated with motivating 
people to answer questions about their future plans/behavior truth- 
fully. In addition, Katona has noted that "Survey research is most 
reliable if it is directed toward obtaining information on variables 
that are widely represented in the universe", [p. 421], agreeing 
that, "unlike laboratory experiments, surveys cannot manipulate 
independent variables" [p. 423] . 

What then would I recommend when it comes right down to the 
question of whether we need, or should use, a survey methodology? 

Based on my experience and education it seems that we have come to 
the juncture of survey and experimental methods. I believe that 
what is required is a damper on the "knee-jerk" propensity to write 
out a survey instrument everytime we need some data and on the equally 
habitual reluctance to incorporate any survey method or data into 
research projects. Our best bet is to integrate, in an innovative 
and appropriate manner, survey methods where the payoff warrants it. 
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The questions we may ask are: 



a) How should this recommended innovative integration of 
survey and non-survey research be determined?; and 

b) How is it being done in economic research at the present? 

In my opinion, the basic question should be, "What am I try- 
ing to do, and what are the feasible ways to do it, and their re- 
spective costs and benefits, both monetary and non-monetary? Each 
of us does this implicitly when we seek data and information for our 
own use. For example, when searching for a place to buy film for 
our camera, we are likely to base our decision on past experience 
and a casual perusal of the newspaper ads. However, when searching 
for a camera, we may ask friends, known to have purchased one, where 
they bought it and if they felt it was a "good buy" . When buying a 
specific make of car, we however tend to rely more heavily on rig- 
orously collected price data, and may only supplement this with a 
friend's opinion. Clearly the greater the cost of making a wrong 
decision, the more accurate and objective we want our data to be. 

The collective experiential survey data of Consumer Reports , cost- 
effectively available, is preferable to a small sample of friends' 
opinions for major purchase decisions. However, we would not indi- 
vidually attempt to collect such data if Consumer Reports was not 
available. It would not be worth it. 

In research projects whose outcome is to be a specific policy 
recommendation, such costs and benefits, in a probablistic sense, 
should be assessed during the research methodology development 
stage. The expected value of each feasible strategy will depend on 
an assessment of the relative accuracy and reliability of the data 
to be so obtained. These factors must be explicitly considered. 
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There is no way to generalize a specific algorithm in such cases 
because there are innumerable unique potential situations. How- 
ever, below I describe a couple of examples of recent research pro- 
jects which illustrate my points. 

Ill . Innovative Use of Modified Survey Methodologies in Economic 
Research 

As mentioned earlier, there is a major problem in using 
survey research in cases where those surveyed have an inherent in- 
centive to hide their true feelings or opinions (as contrasted with 
situations where there are merely insufficient incentives to assess 
them or where significant uncertainty exists) . Thus if in voting 
on a bond issue we incorporated a space for those who vote "yes" 
to indicate how much they would be willing to pay toward the cost of 
providing the facility under consideration, we would expect that 
individuals who favor the issue would either vote "no" hoping that 
enough others would vote "yes" and be assessed, or would vote "for" 
the issue but significantly understate the value they place on having 
the facility hoping to minimize the amount of additional tax they 
personally would have to pay. 

This phenomena has caused great difficulty in the past in 
that we collectively have had to rely upon indirect assessments — 
i.e., everyone's tax would go up (if approved) by the same marginal 
increment. This results in some public goods not being provided 
even though a sufficient number of voters might be willing to pay 
differing amounts to finance it, because a majority were not willing 
to pay the additional stated tax increment. Likewise, even when 
bond issues pass, all this indicates is that enough voters were 
willing to pay the stated "price" , not that the optimal relative tax 
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burden has been defined. 



When voting or otherwise expressing preferences for non-local 
public goods, most times there is little direct connection between 
passage of such an issue and the cost to the voter. Thus, the 
voters or respondents may tend to indicate they favor the adoption 
of some policy because they perceive the cost to them to be negli- 
gible, or fail to perceive the opportunity cost of passage. 

Thus research into the desirability of producing or providing 
such public goods has concentrated on the technological aspects and 
the associated costs of producing the goods or services and to a 
lessor extent on "a priori" utility assessments using utility theory 
and revealed preference. The benefits and opportunity cost asses- 
sments tend to be left to be debated by opposing sides and are 
assumed to be communicated to the population of voters by the media. 
The consumer opinions are then "collected" either through monitoring 
constituent mail or by allowing an eventual popular vote on the 
issue, on an either/or basis only. Although polls may be taken to 
assess the general climate of consumer opinion, the polls are not 
used (generally) to determine the policy decision directly. 

Groves and Ledyard have developed a research methodology [2.] 
which is posited to be a useful first step in changing this process. 
They have designed a "preference revelation" process, with feedback 
to the "voter", which elicits the subjects valuation of the project 
under consideration. The major element present here is a penalty/ 
reward structure for the voter as a result of an indicated decision 
or value assessment. While the details are too complex to present 
here, the salient point for our purposes is that, recognizing the 
inherent difficulties in using a standard survey for value assessment, 
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they have attempted to construct an incentive mechanism tied to 



truthfulness on the survey form to ensure greater accuracy of the 
data collected for decision-making. The results of such a perfected 
methodology would facilitate a benefit calculation to accompany the 
cost estimates of specific public good type projects and would 
allow those projects to be chosen which yield maximum benefits as 
revealed by the population and which would be financed in a pre- 
determined non-homogenous way. 

Groves has also related this "Team Theory" work to the problem 
of managing a decentralized firm [1.]. It appears that this same 
sort of quasi-survey approach with incentives may well have applica- 
tion to the management of the Military Health Care Delivery System 
under Capitation Budgeting with accompanying structural changes. 

The potential for the adoption of a universal enfranchisement 
plan — i.e.. National Health Insurance (NHI) — to significantly 
alter the existing utilization patterns by individuals of the overall 
Health Care Delivery System has made it difficult to estimate 
accurately the cost of the various competing NHI proposals before 
the congress. Our experience with the rapid cost escalation ac- 
companying the adoption of Medicare and Medicaid has caused signifi- 
cant concern and motivated a desire to know more exactly the probable 
financial consequences of NHI implementation. 

Existing (natural experimental) data is not sufficient to 
accomplish these projections because of the significant differences 
in the coverages and real prices of health care to individuals under 
NHI vis-a-vis the current patterns of insurance and care. Likewise, 
the intricate nature of the decision process which leads to the 
observation of a demand for care and the previously discussed 
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"public good" aspect associated with NHI [i.e., if NHI passes, 
everyone has it not just those who might be willing to pay] are 
among the many reasons why a straight-forward survey of individuals 
to ask their probable future utilization under various plans is an 
inappropriate method to provide answers. 

The Rand NHI study, a $32 million project to measure and 
predict the effects of price changes on utilization of health care 
services was thus designed as a mix of an experimental, randomized 
clinical trails type strategy and a survey component. This measure- 
ment of the price effect is not simple. "Measurement of the effect 
of price on demand can be refined in several ways. For example, we 
may wish to know whether the effect of price varies with income and 
total expenditure. Do poor families respond more readily to price 
than affluent ones? If so, they will benefit differentially from a 
generous health insurance plan. Do families with major illnesses 
respond differently from those with minor illnesses? If so, esti- 
mates of the effect of catastrophic health insurance must be based 
only on data from those with such illnesses" [4., p. 6] . Further, 
given that we are interested in effects of price changes, changes 
in coverage must also be evaluated. "To compute this change, one 
must know the current coverage of the population; unfortunately, 
detailed information on this subject is rare. National Health 
Survey estimates can be used to determine the percentage of the 
population with no coverage, but they are not helpful if one wants 
to determine the improvement that would be caused by any particular 
plan .... among persons who now have partial coverage (as most do)" 
[4., p. 6]. The Rand study had to fill these gaps, since, for 
example so few of the respondents know the details of their 
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policy that the National Health Survey has stopped asking such 
questions" [4., p. 9]. 

After painstaking analysis, the Rand group designed a 
5 year experiment in which 3200 families from 4 areas will be 
randomly enrolled in variously structured health plans which 
systematically vary the net price for health care they face. 

Their utilization and expenditure experiences will be carefully 
and exhaustively monitored ($5 million' of the budget is for 
computer charges) . Although most of the data will be collected 
through non-survey methods, surveys do play a significant, and 
integral role in the study. 

For example, in the participant selection process, 6,000 
families from each site, selected from clustered random population 
samples were given screening interview surveys. Subsequently 
2,000 families were chosen for lengthy personal surveys in accord- 
ance with the Morris Finite Selection Model on the basis of the 
information gained from the initial screens. Then, based on the 
more detailed family specific data gathered in the lengthy surveys, 
800 families were selected — 500 for enrollment in the experimental 
group, and 300 for the control group using the Conlisk-Watts Model 
to choose and distribute the families among the various plans to 
minimize non-random variation in the utilization experiences to 
be observed. Finally, quarterly surveys of the participants will 
provide supplemental and valuation data to that collected directly 
during the experiment. 

Although there have been many questions raised about 
specific aspects of the study, it seems a fair evaluation to state 
that the study group, on the basis of rigorous analysis of the 
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potential biases and other problems with the data likely to result, 
has taken a significant step toward providing the most useful demand 
data yet available, as well as providing an example of an innovative 
mix of our two subject research methodologies. 

IV. My View of the Future 

It seems clear to me that the most valuable empirical research 
in the future will utilize a judicious combination of survey and 
experimental methodologies. The character of the major social 
economic problems facing us and the fact that in our ever more 
fine-tuned economy mistakes may impose long term penalties we can 
ill afford to pay, seems unambiguously to point to economic research 
which integrates the probable reactions of consumers to a greater 
extent than ever before. The previous predilection of economists 
(and others) to see the problem in a "partial equilibrium" setting 
is slowly giving way to a broader perspective in structuring research 
to solve seemingly micro scale problems. This will not always in- 
volve survey research, and in fact in many instances, may involve 
less (or no) such methods in cases where they previously had been 
used. But certainly survey research will provide significant bene- 
fits, when carefully tailored to the problem at hand. 

A final example may best illustrate my point. Without going 
into lengthy detail, many, including myself, have recommended that 
the Military Health Services System be significantly restructured 
to decentralize decision making and provide greater incentives for 
cost containment. The major elements of such a system are the use 
of capitated budgets for facilities and their providers, increased 
authority for local decision makers, regional budgets and resource 
responsibility, sharing of the fruits of cost savings with health 
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care providers and staffs, and (eventual) dual choice of enrollment 
for the eligible population. The view of the problem taken is thus 
clearly holistic and any recommendations for a specific implementa- 
tion strategy and associated pilot tests involves many facets. In 
particular we recognized that allowing "dual choice" could have 
potentially catastrophic fiscal implications if many people decided 
to use non-direct facilities and providers (i.e., through CHAMPUS) 
as their source of care. Likewise, we realized that giving the pro- 
viders positive incentives for cost control as we suggested, could 
lead to less acceptable care as evaluated by patients, and thus 
further exacerbate the outflow of patients to CHAMPUS enrollment. 

It was thus deemed important to track the collective patient as- 
sessment of the health care delivery process. This information is 
possessed only by the patient and is by nature extremely subjective. 
Thus the only way to obtain this data was to ask the patients. For 
this reason we have suggested and developed a Patient Satisfaction 
^o^itoring System. This relies on a random sample encounter —based 
survey methodology which provides feedback to the clinic level staff 
and indicates areas in which modifications in their behavior might 
be productive. This information, in combination with the internal 
systemic incentive changes we have suggested, will tend to mitigate 
unnecessary enrollment turnover and hence minimize system costs. 

The point is that the decision to develop and use such a 
survey methodology was made after an analysis of the overall situation 
indicated that it was the best way to obtain the patient— sub jective 
satisfaction data which could have drastically affected system— costs , 
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and hence had to be addressed. Thus, sometimes there are better 
alternatives to the use of surveys, sometimes not. The choice of 
the tool should be dictated by the nature of the problem — not by 
habit or fiat. 
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